1. Field of the Invention
The invention relates generally to image processing and, more specifically, to a system and method for image demosaicing.
2. Description of the Related Art
Digital camera technology has developed rapidly over the past decade. A broad variety of digital cameras are now available to meet the diverse needs of a wide spectrum of consumers. Digital camera technology provides a user with a convenient device to capture image data and to transfer the captured image data to a medium. The medium may be a computer-readable medium, such as a Digital Video Disc (DVD) or a computer memory, or the medium may be a physical medium, such as photographic paper.
In a typical digital camera, a lens projects an image onto a digital sensor that is divided into a grid of pixels. A captured image is a recording of how much light energy is measured at each pixel location. To record color information, a red color filter, a green color filter, or a blue color filter is placed over each pixel of the digital sensor. Many digital sensors are limited to having only one of the three color filters placed over each pixel location. Thus, the sensor is only capable of determining a color value for one of the three red, green, and blue (RGB) colors at each pixel location. For example, if a red color filter is placed over a particular pixel, then the digital sensor measures how much energy is present in the red range of the spectrum at that particular pixel location. Color filters are typically placed over the pixels of the digital sensor in an alternating arrangement known as a Bayer pattern. FIG. 2 is a conceptual diagram that illustrates a typical Bayer pattern, according to the prior art. As shown, a sensor 202 is divided into pixels 210 arranged in a grid. A blue color filter 204, a red color filter 206, or a green color filter 206 is placed over each pixel 210. When the image is captured, only one color component is determined at each pixel location based on which color filter is placed over that pixel.
However, to display the image properly, all three color components are needed at each pixel location. The unknown color components are typically interpolated from the information that is known, i.e., from the surrounding pixels. The interpolation process is known as “demosaicing.” Some conventional algorithms interpolate a missing color component in one of two ways—either horizontally or vertically. A missing color component is either inferred from pixels to the left and right (horizontal interpolation) or from pixels above and below (vertical interpolation). These conventional algorithms then choose which interpolation direction achieves the best result for a particular pixel. For example, if a pixel is located along a horizontal edge in the image, then a typical prior art algorithm would choose to interpolate the missing color component values of the pixel using horizontal interpolation since pixel colors to the left and right (i.e., along the edge) are better indications of what the missing color values should be, compared to the color values of the pixels above and below the edge.
Using some conventional demosaicing approaches, a first interpolated image is generated by interpolating each pixel using horizontal interpolation. Then, a second interpolated image is generated by interpolating each pixel using vertical interpolation. For each pixel location, a selection algorithm then chooses between the interpolated pixel value in the first interpolated image at the pixel location and the interpolated pixel value in the second interpolated image at the pixel location. In general, an implementation does not need to interpolate the entire image plane in a particular direction. Rather, a selection algorithm can determine an interpolation direction for each pixel based on a few neighbors, without the need to interpolate the entire image plane in a particular direction.
To decide which interpolation direction to select for a particular pixel location, a selection algorithm chooses one interpolation direction (either horizontal or vertical) at that particular pixel location. Common methods for interpolation direction selection are based on relative magnitudes of horizontal and vertical gradients. For example, a region of the input image may represented by the following array:
B1G2B3G4R5G6B7G8B9,where B1, B3, B7, and B9 indicate that the blue color value is known at those relative pixel locations, G2, G4, G6, and G8 indicate that the green color value is known at those relative pixel locations, and R5 indicates that the red color value is known at that relative pixel location. To determine the green color value G5 using gradients at position marked by R5, one prior art approach may set G5 to be G5=(G4+G6)/2 if |G4−G6|<|G2−G8|, and let G5=(G2+G8)/2 otherwise.
Other prior art approaches for interpolation direction selection are based on a concept of “homogeneity.” These algorithms define an approximately circular “neighborhood” around the pixel location in question and compute a measure of how much the colors of the neighbors deviate from the color values of the pixel in question based on the interpolation direction. Some conventional algorithms define a threshold amount on a deviation scale and count how many neighbors deviate by less than the threshold amount for each interpolation direction. The number of neighbors that deviate less than the threshold amount for a particular interpolation direction defines a “homogeneity factor” at that pixel location for that interpolation direction. If more neighbors are within the threshold amount when horizontal interpolation is used, then the pixel is interpolated horizontally. Otherwise, the pixel is interpolated vertically. Thus, the algorithm chooses the interpolation direction that yields the higher homogeneity.
FIG. 3A is a conceptual diagram of interpolation regions, according to the prior art. As shown, the interpolation regions include a horizontal interpolation region 302 and a vertical interpolation region 304. Using the horizontal interpolation region 302, pixel color values are interpolated from pixel color values of pixels to the left and to the right of a pixel in question 306. Similarly, using the vertical interpolation region 304, pixel color values are interpolated from pixel color values above and below a pixel in question 308. FIG. 3B is a conceptual diagram of homogeneity neighborhoods, according to the prior art. As shown, an approximately “circular” region 312 surrounding the pixel in question 310 is used to determine the homogeneity of a particular interpolation direction at the pixel in question 310. As shown, the neighborhood may have a radius of one pixel from the pixel in question 310 (e.g., homogeneity neighborhood 314) or a radius of two pixels from the pixel in question 310 (e.g., homogeneity neighborhood 316). In some prior art approaches, the demosaicing algorithm achieves better results using a larger radius for the homogeneity neighborhood.
In most prior art approaches, interpolation directions, like the ones shown in FIG. 3A, are used to determine only the unknown green color values. Once green color values are interpolated for each pixel location, the missing red and blue color values may be found using a different approach.
One problem with conventional demosaicing approaches is that only two interpolation directions (i.e., horizontal and vertical) are used when interpolating pixel color values. Accordingly, when the image includes a diagonal edge, interpolating using horizontal or vertical interpolation results in “zippering” or “checkerboard” artifacts to be introduced into the resultant image.
A second problem with conventional demosaicing approaches that are homogeneity-based is that homogeneity noise is introduced when the interpolation region is different than the neighborhood used to determine homogeneity. As shown in FIGS. 3A and 3B, the interpolation regions are “linear,” meaning that the pixels considered when interpolating color values are within the same row or column of pixels; whereas, the homogeneity neighborhoods are “circular,” meaning that the pixels considered are within some radius of the pixel in question in all directions. Since at least some of the pixels in the homogeneity neighborhood do not contribute to the final color values of the pixel (i.e., at least one pixel location in the homogeneity neighborhood is not included in the interpolation region), using a substantially circular homogeneity neighborhood introduces noise into the homogeneity map. This noise may cause the conventional demosaicing algorithm to fail to choose the best interpolation direction for a particular pixel, resulting in lower image quality.
Accordingly, there remains a need in the art for an improved demosaicing technique that reduces the artifacts and noise introduced using conventional techniques.